feat(pagination): add cursor-based pagination + client APIs by histrio · Pull Request #112 · histrio/py-couchdb

histrio · 2025-09-30T07:25:06Z

No description provided.

codecov-commenter · 2025-09-30T07:26:15Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 93.54839% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.34%. Comparing base (320ad31) to head (a9fd25e).

Files with missing lines	Patch %	Lines
pycouchdb/client.py	90.00%	3 Missing ⚠️
pycouchdb/pagination.py	94.64%	3 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #112      +/-   ##
==========================================
+ Coverage   88.74%   89.34%   +0.59%     
==========================================
  Files           7        8       +1     
  Lines         622      713      +91     
==========================================
+ Hits          552      637      +85     
- Misses         70       76       +6

Flag	Coverage Δ
unittests	`89.34% <93.54%> (+0.59%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot

Pull Request Overview

This PR adds cursor-based pagination functionality to pycouchdb, implementing both view and Mango query pagination along with client API methods. The main purpose is to provide stable, memory-efficient pagination without the pitfalls of skip-based pagination.

Key changes:

Adds a new pagination.py module with view_pages() and mango_pages() functions for cursor-based pagination
Extends the Database class with find(), view_pages(), and mango_pages() methods
Includes comprehensive test coverage for both unit and integration testing

Reviewed Changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
pycouchdb/pagination.py	Core pagination logic implementing cursor-based pagination for views and Mango queries
pycouchdb/client.py	Database class extensions adding pagination methods and Mango query support
pycouchdb/types.py	Type aliases for pagination-related types
pycouchdb/utils.py	Enhanced UTF-8 decoding with error handling
test/test_pagination.py	Unit tests for pagination functions with mocked dependencies
test/test_database.py	Unit tests for new Database methods
test/integration/test_pagination_integration.py	Comprehensive integration tests with real CouchDB
test/integration/test_integration.py	Additional integration tests for pagination functionality
docs/source/quickstart.rst	Documentation updates explaining pagination features

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-09-30T07:26:16Z

pycouchdb/pagination.py

+        last_row = rows[page_size - 1]
+        startkey = last_row['key']
+        startkey_docid = last_row['id']
+        skip = 1  # Skip the row we used as cursor


[nitpick] The inline comment explains what skip=1 does, but it would be clearer to explain why this is necessary to avoid duplicate results in cursor-based pagination.

Suggested change

skip = 1 # Skip the row we used as cursor

skip = 1 # Skip the row used as the cursor to avoid returning it again (prevents duplicate results in cursor-based pagination)

Copilot · 2025-09-30T07:26:17Z

pycouchdb/pagination.py

+def _prepare_mango_data(selector: Dict[str, Any], params: Dict[str, Any]) -> bytes:
+    """Prepare Mango query data for HTTP request."""
+    data_dict = {
+        'selector': selector,
+        **params
+    }
+    return utils.force_bytes(json.dumps(data_dict))


The _prepare_mango_data function is defined but never used in the code. Consider removing it or adding a comment explaining its intended future use.

Suggested change

def _prepare_mango_data(selector: Dict[str, Any], params: Dict[str, Any]) -> bytes:

"""Prepare Mango query data for HTTP request."""

data_dict = {

'selector': selector,

**params

}

return utils.force_bytes(json.dumps(data_dict))

Copilot · 2025-09-30T07:26:17Z

pycouchdb/client.py

+        :param params: Additional query parameters
+        :returns: Iterator yielding lists of rows for each page
+
+        .. versionadded: 1.17


The docstring format should be .. versionadded:: 1.17 with double colons, not single colon.

Suggested change

.. versionadded: 1.17

.. versionadded:: 1.17

Copilot · 2025-09-30T07:26:17Z

pycouchdb/client.py

+        :param params: Additional query parameters
+        :returns: Iterator yielding lists of documents for each page
+
+        .. versionadded: 1.17


The docstring format should be .. versionadded:: 1.17 with double colons, not single colon.

Suggested change

.. versionadded: 1.17

.. versionadded:: 1.17

…omment

Copilot

Pull Request Overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-09-30T08:27:45Z

pycouchdb/pagination.py

+        last_row = rows[page_size - 1]
+        startkey = last_row['key']
+        startkey_docid = last_row['id']
+        skip = 1  # Skip the row used as the cursor to avoid returning it again (prevents duplicate results in cursor-based pagination)


[nitpick] The comment is overly verbose and could be clearer. Consider simplifying to 'Skip cursor row to avoid duplicates'.

Suggested change

skip = 1 # Skip the row used as the cursor to avoid returning it again (prevents duplicate results in cursor-based pagination)

skip = 1 # Skip cursor row to avoid duplicates

Copilot · 2025-09-30T08:27:45Z

pycouchdb/pagination.py

+        startkey = last_row['key']
+        startkey_docid = last_row['id']


The code uses magic string keys 'key' and 'id' without validation. Consider adding defensive checks or constants for these keys to improve maintainability.

Copilot · 2025-09-30T08:27:46Z

test/test_utils.py

+                # Valid JSON but with invalid UTF-8 sequence in the middle
+                self.content = b'{"key": "value", "invalid": "\xff\xfe"}'


The byte string contains invalid UTF-8 sequences inside a JSON string literal, but this creates invalid JSON, not just invalid UTF-8. The test may not be testing the intended scenario.

Suggested change

# Valid JSON but with invalid UTF-8 sequence in the middle

self.content = b'{"key": "value", "invalid": "\xff\xfe"}'

# Valid JSON, but the value for "invalid" is a string with bytes that are invalid UTF-8

self.content = b'{"key": "value", "invalid": "' + b'\xff\xfe'.decode("latin1").encode("utf-8", "surrogateescape") + b'"}'

Copilot · 2025-09-30T08:27:46Z

test/test_utils.py

+                # Valid JSON with invalid UTF-8 that will be replaced with replacement character
+                self.content = b'{"key": "value", "invalid": "\xff\xfe\x80"}'


Similar to the previous test, this creates invalid JSON syntax rather than testing UTF-8 decode errors. The invalid bytes should be outside the JSON string structure.

histrio added 4 commits September 30, 2025 09:22

feat(pagination): add cursor-based pagination + client APIs

15c8f54

docs(quickstart): document view and Mango pagination

77066dd

fix(utils): handle UnicodeDecodeError when parsing JSON

82e2787

test(pagination): add unit and integration coverage

27fa4f8

histrio requested a review from Copilot September 30, 2025 07:25

Copilot AI reviewed Sep 30, 2025

View reviewed changes

histrio added 4 commits September 30, 2025 09:31

fix(mypy): adjust Database.find return and add __all__ in pagination

b05fa86

docs(client): use correct Sphinx versionadded directive

a1cedde

refactor(pagination): remove unused mango helper and clarify cursor c…

e92077f

…omment

test(utils): cover as_json invalid UTF-8 with replacement chars

7aba7a6

histrio requested a review from Copilot September 30, 2025 08:25

Copilot AI reviewed Sep 30, 2025

View reviewed changes

docs(pagination): warn about unreliable pagination with reduced views

da5587e

histrio added the hacktoberfest-accepted label Oct 26, 2025

fix(pagination): avoid double-encoding view parameters

a9fd25e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(pagination): add cursor-based pagination + client APIs#112

feat(pagination): add cursor-based pagination + client APIs#112
histrio wants to merge 10 commits intomasterfrom
feat/pagination-add-cursor-based-pagination-apis

histrio commented Sep 30, 2025

Uh oh!

codecov-commenter commented Sep 30, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Sep 30, 2025

Uh oh!

Copilot AI Sep 30, 2025

Uh oh!

Copilot AI Sep 30, 2025

Uh oh!

Copilot AI Sep 30, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Sep 30, 2025

Uh oh!

Copilot AI Sep 30, 2025

Uh oh!

Copilot AI Sep 30, 2025

Uh oh!

Copilot AI Sep 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	skip = 1 # Skip the row we used as cursor
	skip = 1 # Skip the row used as the cursor to avoid returning it again (prevents duplicate results in cursor-based pagination)

	skip = 1 # Skip the row used as the cursor to avoid returning it again (prevents duplicate results in cursor-based pagination)
	skip = 1 # Skip cursor row to avoid duplicates

		# Valid JSON but with invalid UTF-8 sequence in the middle
		self.content = b'{"key": "value", "invalid": "\xff\xfe"}'

		# Valid JSON with invalid UTF-8 that will be replaced with replacement character
		self.content = b'{"key": "value", "invalid": "\xff\xfe\x80"}'

Conversation

histrio commented Sep 30, 2025

Uh oh!

codecov-commenter commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Copilot AI Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov-commenter commented Sep 30, 2025 •

edited

Loading